What is tokenization on Bablic?
Bablic employs tokenization to minimize repetitive translation and content bleeding. Tokenization identifies and generalizes common patterns within the text to render the translation applicable in multiple variants of similar phrases. This feature eliminates the need to re-translate phrases that are identical, except for a variable value that may not require translation. For example:
| Source Segment | Tokenized Variant |
|---|---|
| You have 5 items in your cart. | You have {{n}} items in your cart. |
| You have 9 items in your cart. | You have {{n}} items in your cart. |
| You have 12 items in your cart. | You have {{n}} items in your cart. |
In this example, Bablic ignores the variable value in the source segment thereby allowing one translation to cover all possible iterations. While there are obvious time and cost efficiencies to tokenization, there may actually be cases where the localization of certain tokens is required for your business.
To understand the right strategy for your website, let's begin by reviewing the units that are treated as tokens by default on Bablic:
- Numbers
- Time zones
- Times
- Dates
- Currencies
- Measurements
If you have business or regional requirements to localize any of these units, there are a few nuances to understand. Here are some examples of localized tokens that might apply to your use-case:
| English Source Segment | French Translation |
|---|---|
| Our company was incorporated on June 05, 2000. | Notre société a été constituée le 5 juin 2000. |
| This is a 3,000 sq ft house. | Il s'agit d'une maison de 278,70 mètres carrés. |
| Your remaining balance is $100.00 | Votre solde restant est de 89,31 € |
In each example, there are multiple intricacies associated with the localization of the token that can be reduced to 3 basic categories.
1 = Reformatting (e.g. MM-DD-YYYY to DD-MM-YYYY)
2 = Conversion & translation (e.g. 3,000 sq ft house to maison de 278,70 mètres carrés)
3 = Placement of separators (e.g. $100.00 to 89,31 €)
Best practice for each category will vary by language and region, but the general concept can be universally applied. Prior to implementing any solution, you will want to have a good understanding of the nuances of the local markets you are targeting.
What are my options?
There are three approaches to token localization on Bablic:
1 = Disable the default Bablic tokenization logic entirely. If you are a user who does a lot of translation editing on Bablic Studio, you'll want to consider that this option could result in inefficiencies during your review. Edits to a segment containing a variable token would not apply to the same segment with a different token value.
2 = With the exception of measurement and currency conversions, Bablic can be used to localize certain tokens such as dates, times and time zones. Please contact the Bablic support team if you wish to implement one of these custom token approaches.
3 = The best approach to token localization is to set up the required logic on the origin website that Bablic is translating for each market. By doing so, you are in complete control of any custom formats and conversions while maintaining Bablic's default token logic designed to maximize time and cost efficiencies.