Telegram Message Entities: Keep Links and Emoji Intact in Your CRM

When a Telegram enquiry reaches a CRM with the wrong link or a clipped word after an emoji, inspect the text mapping before changing webhook delivery. Telegram's message entities identify ranges using UTF-16 code units. Those ranges belong to the original text or caption, not a cleaned copy. Preserve the source string with its entities, then produce a separate plain-text view and validated link records.

This guide is for the maintainer of a custom Telegram Bot API integration. It is a proposed design and local diagnostic, not a verified built-in 24go.site capability. It assumes the integration already receives messages legitimately. The local example needs Python, not a bot token or a CRM account. It does not test a production connector, render untrusted HTML or authorize collecting unrelated people's data.

Identify the source field before reading a range

The Message object has optional text and entities for text messages. Media captions use caption and caption_entities. An entity list describes its corresponding string. A photo with a caption is not a text message with a mysteriously empty body; choosing only text can discard a valid enquiry before formatting is even considered.

Treat the two pairs as distinct inputs to the same mapper. Record which pair was selected. If your supported message has neither field, record an explicit unsupported or no-text outcome rather than making an empty sales note look successful. Other message shapes require their own schema handling; this procedure is not a complete decoder for every current Bot API message type.

Do not substitute the text of a replied-to message, a forwarded source or a generated summary for the current message body. Those are different pieces of context. If the product needs them, store their role explicitly. Otherwise a manager may read an old quoted price as the customer's current request. Preserve the association between each body and the exact entity list that describes it.

Why an emoji can shift a link

An entity is a typed range inside a string. The MessageEntity contract defines both offset and length in UTF-16 code units. A Python string slice uses Unicode code-point positions instead. These coordinate systems agree for many ordinary characters but differ for characters represented by a UTF-16 surrogate pair, including the grinning face in the example below.

Use the synthetic text 😀 offer. The entity covering offer starts at UTF-16 offset 3 and has length 5. The emoji occupies two code units and the following space occupies one. In Python, directly slicing positions 3 through 8 returns ffer, not the requested word. A parser may return a plausible-looking fragment without raising an error, which makes a silent mapping defect more dangerous than an obvious failure.

Do not generalize this to “every emoji counts as two.” Some visible symbols use one code unit, while a displayed emoji sequence can contain several Unicode code points. What looks like one character to a reader is not a dependable offset unit. Use the documented encoding or a tested library helper; do not maintain a hand-written exception list of emoji.

Keep originals and derived representations separate

Store the accepted source string and its associated entity metadata together under the integration's access and retention rules. Derived fields can include a plain-text note, link records and an optional sanitized rich-text view. The original is the reference for mapping and incident reproduction, not a reason to retain every conversation forever.

Normalization is a separate transformation. Trimming leading spaces, replacing line breaks or applying Unicode normalization can change range coordinates or the text they select. Resolve and validate ranges against the unchanged source first. If a downstream view needs different text, calculate its own positions or omit positions from that view. Never attach the original offsets to a modified string and assume they remain valid.

For an update that changes a message, treat the new body and its new metadata as one version. Do not reuse the old entities with the new text. A customer may alter both the label and destination of a link. Updating only the visible note leaves stale link records even when the UTF-16 arithmetic is correct. Agree separately on whether the CRM should replace a note, append a revision or require review.

A link label is not necessarily its destination

A url entity identifies a URL appearing in the body. A text_link entity has a visible label in the text and a separate optional url field that provides the destination. Saving only the selected label loses the target. Presenting that target as though it were the label can hide what the user actually wrote. Keep both fields and the entity type when your workflow needs links.

Neither a Telegram entity nor a familiar-looking label makes a destination trustworthy. The label “company portal” can point somewhere unrelated. Do not fetch the target automatically while creating a CRM note: fetching untrusted addresses introduces a separate network-security boundary. If previews are required, use an independently reviewed service with destination checks, network restrictions and a clear data policy.

For a plain-text CRM, showing the label and a separately identified destination is often simpler than reconstructing Telegram formatting. For a rich-text CRM, escape ordinary text and build only supported markup through a safe renderer. Do not concatenate customer text into HTML or treat incoming formatting as permission to execute anything. A valid range confirms where an entity points, not whether its contents are safe.

A text_mention can contain a user object according to the API. That is not proof that the sender owns the mentioned account or consented to creating a contact for it. Identity verification and contact collection remain separate decisions. Keep the mapping scope narrow instead of promoting any formatted mention into a new CRM person.

Validate ranges before using them

Require integer coordinates within the UTF-16 representation of the selected source. Reject negative offsets, nonpositive lengths for this mapper, ranges past the end and boundaries that split a surrogate pair. Distinguish invalid metadata from a text encoding error. A replacement character inserted during recovery can conceal which input failed; an explicit review outcome is easier to diagnose.

The following code is a small extraction diagnostic, not a complete Telegram entity validator. It deliberately accepts only one string and one range. It checks exact integer types so Python booleans are not silently accepted as coordinates. Strict decoding rejects a boundary inside the emoji's surrogate pair. The tests are artificial and do not send any messages or follow any URLs.

def entity_text(text, offset, length):
    if not isinstance(text, str):
        raise ValueError("text must be a string")
    if type(offset) is not int or type(length) is not int:
        raise ValueError("coordinates must be integers")
    if offset < 0 or length <= 0:
        raise ValueError("invalid range")
    raw = text.encode("utf-16-le", errors="strict")
    start = offset * 2
    end = (offset + length) * 2
    if end > len(raw):
        raise ValueError("range outside text")
    try:
        return raw[start:end].decode("utf-16-le", errors="strict")
    except UnicodeDecodeError as exc:
        raise ValueError("range splits a surrogate pair") from exc

text = "😀 offer"
assert entity_text(text, 3, 5) == "offer"
assert text[3:8] == "ffer"
for offset, length in [(1, 1), (-1, 1), (3, 6), (True, 1)]:
    try:
        entity_text(text, offset, length)
    except ValueError:
        pass
    else:
        raise AssertionError("invalid range accepted")
print("UTF-16 extraction checks passed")

Save the block as a local Python file and run it in a disposable environment. The expected terminal line is UTF-16 extraction checks passed. An assertion failure or exception means this diagnostic did not pass; do not silently change its expected values. The article's accompanying research artifacts record the actual local execution. There was no production bot or CRM test during this check.

This function does not validate entity types, required type-specific fields, nesting rules, URL schemes or CRM permissions. A string containing an unpaired surrogate can fail during encoding before range decoding; route that error to the same explicit invalid-input handling in a real adapter. Do not advertise this short helper as a complete security boundary.

Use a staged mapping procedure

1. Select a supported message body and its matching entity list. Keep source identity, access scope and message version with the mapping job. Do not use a caption's ranges on a text message or combine entity lists from different versions.
2. Validate the body and metadata before producing derived fields. Resolve ranges in the documented coordinate system. Keep malformed inputs available only through the approved review process; avoid copying full enquiries into general application logs.
3. Produce the plain-text note without removing meaningful customer content. Resolve required links into separate records with labels and destinations. Apply a specific allowlist or review policy for supported entity kinds rather than assuming every new type has an equivalent CRM representation.
4. Construct the CRM payload through its documented API and escaping rules. If the target supports only plain text, send plain text. Do not enable a rich-text mode merely to make a single link clickable.
5. Read back the test record through the authorized CRM interface. Compare the complete note, link labels and targets against the intended representation. An HTTP success response cannot show whether the CRM removed markup or changed line breaks.
6. Record a mapping version and the observed outcome. Keep a rollback path for the adapter change, without sending duplicate notes or overwriting later operator edits. Retry behavior belongs to the operation's delivery design, not to the UTF-16 helper.

Use the separate webhook idempotency guide for repeated delivery and ambiguous CRM writes. Correct range extraction does not prevent a second note after an uncertain timeout. Conversely, deduplicating updates does not repair an incorrect link label in the first note.

Build acceptance cases around meaning

A useful regression set varies the position and kind of source content. Include plain ASCII text, Cyrillic text, an emoji before the entity, several entities in one message and a caption with a link. Add a label whose destination differs from its visible words. These are test fixtures, not real customer conversations or evidence that a deployed connector already passes.

For each case, write down the selected source field, exact source string, entity coordinates and expected extracted fragment. Then specify the intended CRM representation separately. A plain-text-only target may intentionally drop bold presentation while preserving every word. That is a supported conversion, not missing data, if the product owner approved it and the test records the decision.

Negative cases should include an out-of-range endpoint, a surrogate-splitting boundary, missing required destination metadata and an unsupported entity kind. Decide whether each produces a plain-text fallback, manual review or a failed mapping job. A fallback must not quietly imply that links were preserved when only labels survived. Keep the status visible to the operator who relies on the note.

Finally test edits: change the message body so the entity moves, and change a link destination while leaving the label unchanged. Verify that the new representation does not keep stale ranges or an old target. Compare the readback, not just the outgoing JSON. These are proposed integration acceptance tests; the local extraction diagnostic proves only its own artificial cases.

Close the incident at the boundary actually verified

A precise resolution might say: the mapper used Python indices for Telegram UTF-16 ranges; the new mapper passed the local extraction fixture, while CRM readback remains pending. That is more useful than declaring Telegram fixed or the sales pipeline healthy. If the source body was already changed before mapping, correct that ownership error as well as the arithmetic.

Monitor mapping failures by reason and adapter version, with only the minimum identifiers needed for investigation. Do not turn error telemetry into a second archive of private enquiries or link credentials. Before enabling production changes, assign responsibility for unsupported shapes and review access. The goal is a faithful, understandable CRM representation, not a promise to reproduce every Telegram presentation feature.

Sources

• Telegram Bot API: Message, source fields and caption metadata, checked on the editorial date.
• Telegram Bot API: MessageEntity, UTF-16 coordinates and type-specific fields, checked on the editorial date.