Both univocity-html-parser and univocity-parsers support the same annotations to generate java beans from rows parsed from the input. You can follow this tutorial to learn about all supported annotations for both libraries.
All annotations you can use are in package com.univocity.parsers.annotations,
The HTML parser has a couple of additional annotations that handle nested collections and maps of objects, which are showcased in the last two sections of this tutorial.
Let’s get started.
Every attribute of your classes that should be populated with values collected from a given result must be annotated with one of:
@Parsed - to bind the attribute of your class to a field of your results.
@Nested - to handle nested classes when your attribute is of a custom type (i.e a class) that has @Parsed and @Nested attributes.
@Parsed is the core annotation to use and is the only one that actually maps fields in your results to an attribute.
Let’s assume your data has the following structure (shown in a fixed-width format for clarity):
profile_id username followers
123.........theuser........200.......
1...........admin....................
You can create a class such as:
class ProfileByFieldName {
@Parsed(field = "profile_id")
private Long profileId;
@Parsed
private String username;
@Parsed
private int followers;
@Override
public String toString() {
return "ProfileByFieldName{" +
"profileId=" + profileId +
", username='" + username + '\'' +
", followers=" + followers +
'}';
}
}
Which maps attribute names to the corresponding headers of your data. Names are automatically mapped if they are the same in both your data and in your class. In the example above the column profile_id doesn’t match with the attribute name profileId, so we had to explicitly map the field name with @Parsed(field = "profile_id").
Alternatively, you can map your attributes to the position of each column of your input:
class ProfileByFieldPosition {
@Parsed(index = 0)
private Long profileId;
@Parsed(index = 1)
private String username;
@Parsed(index = 2)
private int followers;
@Override
public String toString() {
return "ProfileByFieldPosition{" +
"profileId=" + profileId +
", username='" + username + '\'' +
", followers=" + followers +
'}';
}
}
Lastly, if you intend to use the same class to hold values of different sets of results, with different column names, each attribute can be mapped to multiple possible names:
class ProfileByMultipleFieldNames {
@Parsed(field = {"profile_id", "id"})
private Long profileId;
@Parsed(field = {"username", "user"})
private String username;
@Parsed
private int followers;
@Override
public String toString() {
return "ProfileByMultipleFieldNames{" +
"profileId=" + profileId +
", username='" + username + '\'' +
", followers=" + followers +
'}';
}
}
With multiple field names defined, the input data shown earlier should produce the following instances:
ProfileByMultipleFieldNames{profileId=123, username='theuser', followers=200}
ProfileByMultipleFieldNames{profileId=1, username='admin', followers=0}
A different set of input rows can also be processed:
id user created_at fees type admin stars
123.........theuser........2015-10-30...$12,90..U.....no.....143..
1...........admin..........21/05/2001....$0,00..S.....yes....?....
This will produce the same result except for attribute “followers” which is not present:
ProfileByMultipleFieldNames{profileId=123, username='theuser', followers=0}
ProfileByMultipleFieldNames{profileId=1, username='admin', followers=0}
Let’s have a look again at the following user profile data:
id user created_at fees type admin stars
123.........theuser........2015-10-30...$12,90..U.....no.....143..
1...........admin..........21/05/2001....$0,00..S.....yes....?....
To populate each column to a class attribute we need to somehow handle dates and amounts that may have different formats. We also need to convert “yes” and “no” to a proper boolean value. On top of that there’s a “type” column that could be mapped to an enum:
id user created_at fees type admin stars
123.........theuser........2015-10-30...$12,90..U.....no.....143..
1...........admin..........21/05/2001....$0,00..S.....yes....?....
This actually involves very little work. We can simply define a class such as:
class Profile {
enum Type {
USER('U'),
SYSTEM('S');
public final char typeCode;
Type(char typeCode) {
this.typeCode = typeCode;
}
}
@Parsed
private Long id;
@Parsed
@Trim
@UpperCase
private String user;
@Parsed(field = "created_at")
@Format(formats = {"yyyy-MM-dd", "dd/MM/yyyy"}, options = "locale=en;lenient=false")
private Date createdAt;
@Parsed
@Replace(expression = "\\$", replacement = "")
@Format(formats = {"#0,00"}, options = "decimalSeparator=,")
private BigDecimal fees;
@Parsed
@BooleanString(trueStrings = {"yes", "y"}, falseStrings = {"no", "n"})
private boolean admin;
@Parsed
@EnumOptions(customElement = "typeCode")
private Type type;
@Parsed(field = "stars")
@NullString(nulls = {"?", "N/A"})
private Integer stars;
@Override
public String toString() {
return "Profile{" +
"id=" + id +
", user='" + user + '\'' +
", createdAt=" + createdAt +
", fees=" + fees +
", admin=" + admin +
", type=" + type +
", stars=" + stars +
'}';
}
//
And the objects generated from that input will be:
Profile{id=123, user='THEUSER', createdAt=Fri Oct 30 00:00:00 CST 2015, fees=12.90, admin=false, type=USER, stars=143}
Profile{id=1, user='ADMIN', createdAt=Mon May 21 00:00:00 CST 2001, fees=0.00, admin=true, type=SYSTEM, stars=null}
Let’s explain each annotation:
Trim: trims any whitespaces around a value;
UpperCase: upper cases the value;
LowerCase: lower cases the value;
Replace: performs a String.replace operation on the value, matching the regular expression provided in the expression property and replacing with the replacement string provided. In our example use used it to remove the ‘$’ character from values in the “fees” column.
NullString: converts a String in the input record to null, as it represents a missing value. In our example, values “?” or “N/A” will be converted to null
Format: applies to Date, Calendar or any numeric type attribute and accepts multiple formats. The formats and options to use depend on the attribute type.
with dates the formats you define are handled by Java’s SimpleDateFormat so they must conform to the patterns accepted by that class. The options allow you to configure properties of the SimpleDateFormat that is created internally. You must provide key-value pairs separated by a ;. In the given example “locale=en” will internally execute new SimpleDateFormat("yyyy-MM-dd", new Locale("en")); and “lenient=false” executes simpleDateFormat.setLenient(false);.
DecimalFormat. Similarly to the date patterns, options will set attributes of the DecimalFormat instance used to parse the input values, as well as the associated DecimalFormatSymbols.BooleanString: accepts sets of multiple Strings that represent either true or false. If the records collected by the parser have a value that match with the set of trueStrings, the attribute will be set to true. Otherwise, false will be determined from the set of falseStrings. If neither sets match, DataProcessingException will be raised informing you that no boolean value can be determined.
EnumOptions: converts the plain String in the input record to an enumeration value. Enumerations are mapped automatically if one of the following matches with the input value, in order of precedence:
the name of a value of the enum
the ordinal value (i.e. the input value is a numeric representation of the enumeration’s ordinal);
toString() of a value of the enumThis order can be redefined in the selectors attribute of EnumOptions
In the example we used, the Type enumeration has a typeCode character that represents each human-readable value. The records we are processing use that character, and we can easily map it by setting the customElement in @EnumOptions. The parser will discover what values to use based on the typeCode and map them to the Type values accordingly.
You can create your own implementation of the Conversion interface so it can be used in fields annotated with Parsed.
A varargs constructor can be used on your implementation so it can also be initialized with String... args.
For example, the following custom implementation splits a String into a String[] based on a given delimiter.
class Splitter implements Conversion<String, String[]> {
private String separator;
public Splitter(String... args) {
if(args.length == 0){
separator = ",";
} else {
separator = args[0];
}
}
@Override
public String[] execute(String input) {
if(input == null){
return new String[0];
}
return input.split(separator);
}
@Override
public String revert(String[] input) {
StringBuilder out = new StringBuilder();
for (String value : input) {
if (out.length() > 0) {
out.append(separator);
}
out.append(value);
}
return out.toString();
}
}
Now, assuming the parser produces the following records:
id user words
123.........theuser........cheater;snitch
222.........otheruser..............camper
We can create a class such as:
class Offender {
@Nested
private Profile profile;
@Parsed
@Convert(conversionClass = Splitter.class, args = ";")
private String[] words;
@Override
public String toString() {
return "Offender{" +
"id=" + profile.getId() +
", user='" + profile.getUser() +
"', words=" + Arrays.toString(words) +
'}';
}
}
Which will make use of our custom conversion to split words in the words column based on the ‘;’ separator. Our Offender objects will be populated with:
Offender{id=123, user='THEUSER', words=[cheater, snitch]}
Offender{id=222, user='OTHERUSER', words=[camper]}
We also used the Nested annotation to reuse the existing Profile class. We are going to talk more about it in the next section.
All annotations can be put on the methods of your class instead of directly into its fields. The BetterOffender class below makes the previous Offender example that depends on a custom conversion look silly:
class BetterOffender {
@Nested
private Profile profile;
private String[] words;
@Parsed(field = "words")
public void setWords(String words) {
this.words = words.split(";");
}
@Override
public String toString() {
return "BetterOffender{" +
"id=" + profile.getId() +
", user='" + profile.getUser() +
"', words=" + Arrays.toString(words) +
'}';
}
}
Just keep in mind that if you intend to use the same class for writing, you will want to annotate the “getter” method as well so the attribute value can be read from your objects.
The @Nested annotation works to populate nested types in your class structure from a single input row. This means that the nested attributes have their headers collected and joined together with the headers of the parent class.
Let’s look again at the BetterOffender class:
class BetterOffender {
@Nested
private Profile profile;
private String[] words;
@Parsed(field = "words")
public void setWords(String words) {
this.words = words.split(";");
}
@Override
public String toString() {
return "BetterOffender{" +
"id=" + profile.getId() +
", user='" + profile.getUser() +
"', words=" + Arrays.toString(words) +
'}';
}
}
This class structure is capable to reading the following input columns of a given result:
Profile class (as defined in the annotations in each attribute of Profile)PLUS
BetterOffender class itself (the field words is defined in the annotation of the setWords method)So this input:
id user words
123.........theuser........cheater;snitch
222.........otheruser..............camper
Can be parsed into instances of BetterOffender, whose toString will show:
BetterOffender{id=123, user='THEUSER', words=[cheater, snitch]}
BetterOffender{id=222, user='OTHERUSER', words=[camper]}
In some cases you will want to use the same type of @Nested attribute, for example consider the AddressBook that has two nested Address types:
class AddressBook {
@Nested
private Address mailingAddress;
@Nested
private Address mainAddress;
Where the Address class is simply:
class Address {
@Parsed
private String street;
@Parsed
private String city;
@Parsed
private String state;
@Override
public String toString() {
return "Address{" +
"street='" + street + '\'' +
", city='" + city + '\'' +
", state='" + state + '\'' +
'}';
}
}
The problem here is that mailingAddress and mainAddress have the same type and consequently the same field names. Luckily we can transform the field names used by each nested Address with the help of a HeaderTransformer.
Consider the parsed records have this structure:
mail_street mail_city mail_state main_street main_city main_state
12 Some St.......Sydney............NSW...23 The St........Brisbane..........QLD
450 Other St.....Adelaide...........SA...2 William Av.....Hobart............TAS
As the column names are prefixed with mail for the mailing address and main for the main address, we can implement a HeaderTransformer such as:
class AddressTypeTransformer extends HeaderTransformer {
private String prefix;
public AddressTypeTransformer(String... args) {
prefix = args[0];
}
@Override
public String transformName(Field field, String name) {
return prefix + "_" + name;
}
}
Which will insert a given prefix to the field names in our nested Address attributes. The AddressBook class must be updated to make use of our AddressTypeTransformer:
class AddressBook {
@Nested(headerTransformer = AddressTypeTransformer.class, args = "mail")
private Address mailingAddress;
@Nested(headerTransformer = AddressTypeTransformer.class, args = "main")
private Address mainAddress;
@Override
public String toString() {
return "AddressBook{" +
"mailing=" + mailingAddress +
", main=" + mainAddress +
'}';
}
}
With this, mailingAddress.street will get the values from column mail_street, and mainAddress.street from main_street. The other attributes will be mapped similarly.
When converting the input records to java beans, we get the correct mapping:
AddressBook{mailing=Address{street='12 Some St', city='Sydney', state='NSW'}, main=Address{street='23 The St', city='Brisbane', state='QLD'}}
AddressBook{mailing=Address{street='450 Other St', city='Adelaide', state='SA'}, main=Address{street='2 William Av', city='Hobart', state='TAS'}}
In many cases a class can have multiple attributes that require the usage of the same set of annotations, over and over. For example, an input such as:
id created_at updated_at deleted_at
123..2010-12-20..2018-02-05..2018-04-01
456..2011-01-29..2017-06-26......-.....
All dates are formatted using the yyyy-MM-dd format, or shown as a ‘-’ if the date is not available (i.e. null).
When dealing with this the first approach uninformed people tend to use is to define a class with repetitive annotations. It usually becomes something like this:
class DatesRepetitive {
@Parsed
private Long id;
@Parsed(field = "created_at")
@NullString(nulls = "-")
@Format(formats = "yyyy-MM-dd")
private Date createdAt;
@Parsed(field = "updated_at")
@NullString(nulls = "-")
@Format(formats = "yyyy-MM-dd")
private Date updatedAt;
@Parsed(field = "deleted_at")
@NullString(nulls = "-")
@Format(formats = "yyyy-MM-dd")
private Date deletedAt;
//
Maybe it’s not so bad to add the same 3 annotations to every dated field in your class, but it can get much worse if you need to add the same annotations in multiple classes, and many more fields. To avoid repetition, you can define a meta-annotation instead.
A meta-annotation is your own annotation type which you can use in place of multiple annotations. You can create a @MyCompanyDate annotation type easily and use in your classes to make them look like this:
class DatesWithMetaAnnotation {
@Parsed
private Long id;
@MyCompanyDate(field = "created_at")
private Date createdAt;
@MyCompanyDate(field = "updated_at")
private Date updatedAt;
@MyCompanyDate(field = "deleted_at")
private Date deletedAt;
//
The definition of @MyCompanyDate is relatively straightforward:
//default annotation stuff - you'll probably just want to copy and paste this
@Retention(RetentionPolicy.RUNTIME)
@Inherited
@Target({ElementType.FIELD, ElementType.METHOD, ElementType.ANNOTATION_TYPE})
//your common annotations go here
@Parsed
@NullString(nulls = "-")
@Format(formats = "yyyy-MM-dd")
public @interface MyCompanyDate {
@Copy(to = Parsed.class) //copies the value provided in `MyCompanyDate.field` to `Parsed.field`
String field() default "";
}
So all you have to do is to add the group of annotations you use everywhere to the meta-annotation declaration itself.
We added a field property to the meta-annotation with the intent of allowing you to map attributes to a custom column name of your results. This works exactly like the field property of the @Parsed annotation because the @Copy annotation copies the value set in @MyCompanyDate.field into @Parsed.field.
The Copy annotation can copy any value of any property of your meta-annotation. Where property names are different, you need to provide the target property name. For example, we could add a position attribute to the meta-annotation and copy it’s value to Parsed.index with:
@Copy(to = Parsed.class, property = "index")
int position() default -1;
This is very useful when you find yourself using the same annotations over and over. You can even create meta-annotations using other meta-annotations.
The @Linked annotation is available only to the HTML parser currently, and allows you to populate single objects, arrays, collections and maps of different types from results that are linked to a parent record. The association among records can exist naturally via results collected via Link following), or manually by joining results collected by multiple entities in the same page.
Proceed to Reading linked results into java beans to learn more.