Format Settings
These settings are autogenerated from source.
bool_false_representation
Type: String
Default value: false
Text to represent false bool value in TSV/CSV/Vertical/Pretty formats.
bool_true_representation
Type: String
Default value: true
Text to represent true bool value in TSV/CSV/Vertical/Pretty formats.
column_names_for_schema_inference
Type: String
Default value:
The list of column names to use in schema inference for formats without column names. The format: 'column1,column2,column3,...'
cross_to_inner_join_rewrite
Type: UInt64
Default value: 1
Use inner join instead of comma/cross join if there are joining expressions in the WHERE section. Values: 0 - no rewrite, 1 - apply if possible for comma/cross, 2 - force rewrite all comma joins, cross - if possible
date_time_64_output_format_cut_trailing_zeros_align_to_groups_of_thousands
Type: Bool
Default value: 0
Dynamically trim the trailing zeros of datetime64 values to adjust the output scale to [0, 3, 6], corresponding to 'seconds', 'milliseconds', and 'microseconds'
date_time_input_format
Type: DateTimeInputFormat
Default value: basic
Allows choosing a parser of the text representation of date and time.
The setting does not apply to date and time functions.
Possible values:
-
'best_effort'
— Enables extended parsing.ClickHouse can parse the basic
YYYY-MM-DD HH:MM:SS
format and all ISO 8601 date and time formats. For example,'2018-06-08T01:02:03.000Z'
. -
'basic'
— Use basic parser.ClickHouse can parse only the basic
YYYY-MM-DD HH:MM:SS
orYYYY-MM-DD
format. For example,2019-08-20 10:18:56
or2019-08-20
.
Cloud default value: 'best_effort'
.
See also:
date_time_output_format
Type: DateTimeOutputFormat
Default value: simple
Allows choosing different output formats of the text representation of date and time.
Possible values:
-
simple
- Simple output format.ClickHouse output date and time
YYYY-MM-DD hh:mm:ss
format. For example,2019-08-20 10:18:56
. The calculation is performed according to the data type's time zone (if present) or server time zone. -
iso
- ISO output format.ClickHouse output date and time in ISO 8601
YYYY-MM-DDThh:mm:ssZ
format. For example,2019-08-20T10:18:56Z
. Note that output is in UTC (Z
means UTC). -
unix_timestamp
- Unix timestamp output format.ClickHouse output date and time in Unix timestamp format. For example
1566285536
.
See also:
date_time_overflow_behavior
Type: DateTimeOverflowBehavior
Default value: ignore
Defines the behavior when Date, Date32, DateTime, DateTime64 or integers are converted into Date, Date32, DateTime or DateTime64 but the value cannot be represented in the result type.
Possible values:
ignore
— Silently ignore overflows. Result are undefined.throw
— Throw an exception in case of overflow.saturate
— Saturate the result. If the value is smaller than the smallest value that can be represented by the target type, the result is chosen as the smallest representable value. If the value is bigger than the largest value that can be represented by the target type, the result is chosen as the largest representable value.
Default value: ignore
.
dictionary_use_async_executor
Type: Bool
Default value: 0
Execute a pipeline for reading dictionary source in several threads. It's supported only by dictionaries with local CLICKHOUSE source.
errors_output_format
Type: String
Default value: CSV
Method to write Errors to text output.
exact_rows_before_limit
Type: Bool
Default value: 0
When enabled, ClickHouse will provide exact value for rows_before_limit_at_least statistic, but with the cost that the data before limit will have to be read completely
format_avro_schema_registry_url
Type: URI
Default value:
For AvroConfluent format: Confluent Schema Registry URL.
format_binary_max_array_size
Type: UInt64
Default value: 1073741824
The maximum allowed size for Array in RowBinary format. It prevents allocating large amount of memory in case of corrupted data. 0 means there is no limit
format_binary_max_string_size
Type: UInt64
Default value: 1073741824
The maximum allowed size for String in RowBinary format. It prevents allocating large amount of memory in case of corrupted data. 0 means there is no limit
format_capn_proto_enum_comparising_mode
Type: CapnProtoEnumComparingMode
Default value: by_values
How to map ClickHouse Enum and CapnProto Enum
format_capn_proto_use_autogenerated_schema
Type: Bool
Default value: 1
Use autogenerated CapnProto schema when format_schema is not set
format_csv_allow_double_quotes
Type: Bool
Default value: 1
If it is set to true, allow strings in double quotes.
format_csv_allow_single_quotes
Type: Bool
Default value: 0
If it is set to true, allow strings in single quotes.
format_csv_delimiter
Type: Char
Default value: ,
The character to be considered as a delimiter in CSV data. If setting with a string, a string has to have a length of 1.
format_csv_null_representation
Type: String
Default value: \N
Custom NULL representation in CSV format
format_custom_escaping_rule
Type: EscapingRule
Default value: Escaped
Field escaping rule (for CustomSeparated format)
format_custom_field_delimiter
Type: String
Default value:
Delimiter between fields (for CustomSeparated format)
format_custom_result_after_delimiter
Type: String
Default value:
Suffix after result set (for CustomSeparated format)
format_custom_result_before_delimiter
Type: String
Default value:
Prefix before result set (for CustomSeparated format)
format_custom_row_after_delimiter
Type: String
Default value:
Delimiter after field of the last column (for CustomSeparated format)
format_custom_row_before_delimiter
Type: String
Default value:
Delimiter before field of the first column (for CustomSeparated format)
format_custom_row_between_delimiter
Type: String
Default value:
Delimiter between rows (for CustomSeparated format)
format_display_secrets_in_show_and_select
Type: Bool
Default value: 0
Enables or disables showing secrets in SHOW
and SELECT
queries for tables, databases,
table functions, and dictionaries.
User wishing to see secrets must also have
display_secrets_in_show_and_select
server setting
turned on and a
displaySecretsInShowAndSelect
privilege.
Possible values:
- 0 — Disabled.
- 1 — Enabled.
format_json_object_each_row_column_for_object_name
Type: String
Default value:
The name of column that will be used for storing/writing object names in JSONObjectEachRow format.
Column type should be String. If value is empty, default names row_{i}
will be used for object names.
input_format_json_compact_allow_variable_number_of_columns
Allow variable number of columns in rows in JSONCompact/JSONCompactEachRow input formats. Ignore extra columns in rows with more columns than expected and treat missing columns as default values.
Disabled by default.
output_format_markdown_escape_special_characters
When enabled, escape special characters in Markdown.
Common Mark defines the following special characters that can be escaped by :
! " # $ % & ' ( ) * + , - . / : ; < = > ? @ [ \ ] ^ _ ` { | } ~
Possible values:
- 0 — Disable.
- 1 — Enable.
input_format_json_empty_as_default
When enabled, replace empty input fields in JSON with default values. For complex default expressions input_format_defaults_for_omitted_fields
must be enabled too.
Possible values:
- 0 — Disable.
- 1 — Enable.
format_protobuf_use_autogenerated_schema
Type: Bool
Default value: 1
Use autogenerated Protobuf when format_schema is not set
format_regexp
Type: String
Default value:
Regular expression (for Regexp format)
format_regexp_escaping_rule
Type: EscapingRule
Default value: Raw
Field escaping rule (for Regexp format)
format_regexp_skip_unmatched
Type: Bool
Default value: 0
Skip lines unmatched by regular expression (for Regexp format)
format_schema
Type: String
Default value:
This parameter is useful when you are using formats that require a schema definition, such as Cap’n Proto or Protobuf. The value depends on the format.
format_template_resultset
Type: String
Default value:
Path to file which contains format string for result set (for Template format)
format_template_resultset_format
Type: String
Default value:
Format string for result set (for Template format)
format_template_row
Type: String
Default value:
Path to file which contains format string for rows (for Template format)
format_template_row_format
Type: String
Default value:
Format string for rows (for Template format)
format_template_rows_between_delimiter
Type: String
Default value:
Delimiter between rows (for Template format)
format_tsv_null_representation
Type: String
Default value: \N
Custom NULL representation in TSV format
input_format_allow_errors_num
Type: UInt64
Default value: 0
Sets the maximum number of acceptable errors when reading from text formats (CSV, TSV, etc.).
The default value is 0.
Always pair it with input_format_allow_errors_ratio
.
If an error occurred while reading rows but the error counter is still less than input_format_allow_errors_num
, ClickHouse ignores the row and moves on to the next one.
If both input_format_allow_errors_num
and input_format_allow_errors_ratio
are exceeded, ClickHouse throws an exception.
input_format_allow_errors_ratio
Type: Float
Default value: 0
Sets the maximum percentage of errors allowed when reading from text formats (CSV, TSV, etc.). The percentage of errors is set as a floating-point number between 0 and 1.
The default value is 0.
Always pair it with input_format_allow_errors_num
.
If an error occurred while reading rows but the error counter is still less than input_format_allow_errors_ratio
, ClickHouse ignores the row and moves on to the next one.
If both input_format_allow_errors_num
and input_format_allow_errors_ratio
are exceeded, ClickHouse throws an exception.
input_format_allow_seeks
Type: Bool
Default value: 1
Allow seeks while reading in ORC/Parquet/Arrow input formats.
Enabled by default.
input_format_arrow_allow_missing_columns
Type: Bool
Default value: 1
Allow missing columns while reading Arrow input formats
input_format_arrow_case_insensitive_column_matching
Type: Bool
Default value: 0
Ignore case when matching Arrow columns with CH columns.
input_format_arrow_skip_columns_with_unsupported_types_in_schema_inference
Type: Bool
Default value: 0
Skip columns with unsupported types while schema inference for format Arrow
input_format_avro_allow_missing_fields
Type: Bool
Default value: 0
For Avro/AvroConfluent format: when field is not found in schema use default value instead of error
input_format_avro_null_as_default
Type: Bool
Default value: 0
For Avro/AvroConfluent format: insert default in case of null and non Nullable column
input_format_binary_decode_types_in_binary_format
Type: Bool
Default value: 0
Read data types in binary format instead of type names in RowBinaryWithNamesAndTypes input format
input_format_binary_read_json_as_string
Type: Bool
Default value: 0
Read values of JSON data type as JSON String values in RowBinary input format.
input_format_bson_skip_fields_with_unsupported_types_in_schema_inference
Type: Bool
Default value: 0
Skip fields with unsupported types while schema inference for format BSON.
input_format_capn_proto_skip_fields_with_unsupported_types_in_schema_inference
Type: Bool
Default value: 0
Skip columns with unsupported types while schema inference for format CapnProto
input_format_csv_allow_cr_end_of_line
Type: Bool
Default value: 0
If it is set true, \r will be allowed at end of line not followed by \n
input_format_csv_allow_variable_number_of_columns
Type: Bool
Default value: 0
Ignore extra columns in CSV input (if file has more columns than expected) and treat missing fields in CSV input as default values
input_format_csv_allow_whitespace_or_tab_as_delimiter
Type: Bool
Default value: 0
Allow to use spaces and tabs(\t) as field delimiter in the CSV strings
input_format_csv_arrays_as_nested_csv
Type: Bool
Default value: 0
When reading Array from CSV, expect that its elements were serialized in nested CSV and then put into string. Example: "[""Hello"", ""world"", ""42"""" TV""]". Braces around array can be omitted.
input_format_csv_deserialize_separate_columns_into_tuple
Type: Bool
Default value: 1
If it set to true, then separate columns written in CSV format can be deserialized to Tuple column.
input_format_csv_detect_header
Type: Bool
Default value: 1
Automatically detect header with names and types in CSV format
input_format_csv_empty_as_default
Type: Bool
Default value: 1
Treat empty fields in CSV input as default values.
input_format_csv_enum_as_number
Type: Bool
Default value: 0
Treat inserted enum values in CSV formats as enum indices
input_format_csv_skip_first_lines
Type: UInt64
Default value: 0
Skip specified number of lines at the beginning of data in CSV format
input_format_csv_skip_trailing_empty_lines
Type: Bool
Default value: 0
Skip trailing empty lines in CSV format
input_format_csv_trim_whitespaces
Type: Bool
Default value: 1
Trims spaces and tabs (\t) characters at the beginning and end in CSV strings
input_format_csv_try_infer_numbers_from_strings
Type: Bool
Default value: 0
If enabled, during schema inference ClickHouse will try to infer numbers from string fields. It can be useful if CSV data contains quoted UInt64 numbers.
Disabled by default.
input_format_csv_try_infer_strings_from_quoted_tuples
Type: Bool
Default value: 1
Interpret quoted tuples in the input data as a value of type String.
input_format_csv_use_best_effort_in_schema_inference
Type: Bool
Default value: 1
Use some tweaks and heuristics to infer schema in CSV format
input_format_csv_use_default_on_bad_values
Type: Bool
Default value: 0
Allow to set default value to column when CSV field deserialization failed on bad value
input_format_custom_allow_variable_number_of_columns
Type: Bool
Default value: 0
Ignore extra columns in CustomSeparated input (if file has more columns than expected) and treat missing fields in CustomSeparated input as default values
input_format_custom_detect_header
Type: Bool
Default value: 1
Automatically detect header with names and types in CustomSeparated format
input_format_custom_skip_trailing_empty_lines
Type: Bool
Default value: 0
Skip trailing empty lines in CustomSeparated format
input_format_defaults_for_omitted_fields
Type: Bool
Default value: 1
When performing INSERT
queries, replace omitted input column values with default values of the respective columns. This option applies to JSONEachRow (and other JSON formats), CSV, TabSeparated, TSKV, Parquet, Arrow, Avro, ORC, Native formats and formats with WithNames
/WithNamesAndTypes
suffixes.
When this option is enabled, extended table metadata are sent from server to client. It consumes additional computing resources on the server and can reduce performance.
Possible values:
- 0 — Disabled.
- 1 — Enabled.
input_format_force_null_for_omitted_fields
Type: Bool
Default value: 0
Force initialize omitted fields with null values
input_format_hive_text_allow_variable_number_of_columns
Type: Bool
Default value: 1
Ignore extra columns in Hive Text input (if file has more columns than expected) and treat missing fields in Hive Text input as default values
input_format_hive_text_collection_items_delimiter
Type: Char
Default value:
Delimiter between collection(array or map) items in Hive Text File
input_format_hive_text_fields_delimiter
Type: Char
Default value:
Delimiter between fields in Hive Text File
input_format_hive_text_map_keys_delimiter
Type: Char
Default value:
Delimiter between a pair of map key/values in Hive Text File
input_format_import_nested_json
Type: Bool
Default value: 0
Enables or disables the insertion of JSON data with nested objects.
Supported formats:
Possible values:
- 0 — Disabled.
- 1 — Enabled.
See also:
- Usage of Nested Structures with the
JSONEachRow
format.
input_format_ipv4_default_on_conversion_error
Type: Bool
Default value: 0
Deserialization of IPv4 will use default values instead of throwing exception on conversion error.
Disabled by default.
input_format_ipv6_default_on_conversion_error
Type: Bool
Default value: 0
Deserialization of IPV6 will use default values instead of throwing exception on conversion error.
Disabled by default.
input_format_json_compact_allow_variable_number_of_columns
Type: Bool
Default value: 0
Ignore extra columns in JSONCompact(EachRow) input (if file has more columns than expected) and treat missing fields in JSONCompact(EachRow) input as default values
input_format_json_defaults_for_missing_elements_in_named_tuple
Type: Bool
Default value: 1
Insert default values for missing elements in JSON object while parsing named tuple.
This setting works only when setting input_format_json_named_tuples_as_objects
is enabled.
Enabled by default.
input_format_json_empty_as_default
Type: Bool
Default value: 0
Treat empty fields in JSON input as default values.
input_format_json_ignore_unknown_keys_in_named_tuple
Type: Bool
Default value: 1
Ignore unknown keys in json object for named tuples.
Enabled by default.
input_format_json_ignore_unnecessary_fields
Type: Bool
Default value: 1
Ignore unnecessary fields and not parse them. Enabling this may not throw exceptions on json strings of invalid format or with duplicated fields
input_format_json_infer_incomplete_types_as_strings
Type: Bool
Default value: 1
Allow to use String type for JSON keys that contain only Null
/{}
/[]
in data sample during schema inference.
In JSON formats any value can be read as String, and we can avoid errors like Cannot determine type for column 'column_name' by first 25000 rows of data, most likely this column contains only Nulls or empty Arrays/Maps
during schema inference
by using String type for keys with unknown types.
Example:
SET input_format_json_infer_incomplete_types_as_strings = 1, input_format_json_try_infer_named_tuples_from_objects = 1;
DESCRIBE format(JSONEachRow, '{"obj" : {"a" : [1,2,3], "b" : "hello", "c" : null, "d" : {}, "e" : []}}');
SELECT * FROM format(JSONEachRow, '{"obj" : {"a" : [1,2,3], "b" : "hello", "c" : null, "d" : {}, "e" : []}}');
Result:
┌─name─┬─type───────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬─default_type─┬─default_expression─┬─comment─┬─codec_expression─┬─ttl_expression─┐
│ obj │ Tuple(a Array(Nullable(Int64)), b Nullable(String), c Nullable(String), d Nullable(String), e Array(Nullable(String))) │ │ │ │ │ │
└──────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴──────────────┴────────────────────┴─────────┴──────────────────┴────────────────┘
┌─obj────────────────────────────┐
│ ([1,2,3],'hello',NULL,'{}',[]) │
└────────────────────────────────┘
Enabled by default.
input_format_json_max_depth
Type: UInt64
Default value: 1000
Maximum depth of a field in JSON. This is not a strict limit, it does not have to be applied precisely.
input_format_json_named_tuples_as_objects
Type: Bool
Default value: 1
Parse named tuple columns as JSON objects.
Enabled by default.
input_format_json_read_arrays_as_strings
Type: Bool
Default value: 1
Allow parsing JSON arrays as strings in JSON input formats.
Example:
SET input_format_json_read_arrays_as_strings = 1;
SELECT arr, toTypeName(arr), JSONExtractArrayRaw(arr)[3] from format(JSONEachRow, 'arr String', '{"arr" : [1, "Hello", [1,2,3]]}');
Result:
┌─arr───────────────────┬─toTypeName(arr)─┬─arrayElement(JSONExtractArrayRaw(arr), 3)─┐
│ [1, "Hello", [1,2,3]] │ String │ [1,2,3] │
└───────────────────────┴─────────────────┴───────────────────────────────────────────┘
Enabled by default.
input_format_json_read_bools_as_numbers
Type: Bool
Default value: 1
Allow parsing bools as numbers in JSON input formats.
Enabled by default.
input_format_json_read_bools_as_strings
Type: Bool
Default value: 1
Allow parsing bools as strings in JSON input formats.
Enabled by default.
input_format_json_read_numbers_as_strings
Type: Bool
Default value: 1
Allow parsing numbers as strings in JSON input formats.
Enabled by default.
input_format_json_read_objects_as_strings
Type: Bool
Default value: 1
Allow parsing JSON objects as strings in JSON input formats.
Example:
SET input_format_json_read_objects_as_strings = 1;
CREATE TABLE test (id UInt64, obj String, date Date) ENGINE=Memory();
INSERT INTO test FORMAT JSONEachRow {"id" : 1, "obj" : {"a" : 1, "b" : "Hello"}, "date" : "2020-01-01"};
SELECT * FROM test;
Result:
┌─id─┬─obj──────────────────────┬───────date─┐
│ 1 │ {"a" : 1, "b" : "Hello"} │ 2020-01-01 │
└────┴──────────────────────────┴────────────┘
Enabled by default.
input_format_json_throw_on_bad_escape_sequence
Type: Bool
Default value: 1
Throw an exception if JSON string contains bad escape sequence in JSON input formats. If disabled, bad escape sequences will remain as is in the data.
Enabled by default.
input_format_json_try_infer_named_tuples_from_objects
Type: Bool
Default value: 1
If enabled, during schema inference ClickHouse will try to infer named Tuple from JSON objects. The resulting named Tuple will contain all elements from all corresponding JSON objects from sample data.
Example:
SET input_format_json_try_infer_named_tuples_from_objects = 1;
DESC format(JSONEachRow, '{"obj" : {"a" : 42, "b" : "Hello"}}, {"obj" : {"a" : 43, "c" : [1, 2, 3]}}, {"obj" : {"d" : {"e" : 42}}}')
Result:
┌─name─┬─type───────────────────────────────────────────────────────────────────────────────────────────────┬─default_type─┬─default_expression─┬─comment─┬─codec_expression─┬─ttl_expression─┐
│ obj │ Tuple(a Nullable(Int64), b Nullable(String), c Array(Nullable(Int64)), d Tuple(e Nullable(Int64))) │ │ │ │ │ │
└──────┴────────────────────────────────────────────────────────────────────────────────────────────────────┴──────────────┴────────────────────┴─────────┴──────────────────┴────────────────┘
Enabled by default.
input_format_json_try_infer_numbers_from_strings
Type: Bool
Default value: 0
If enabled, during schema inference ClickHouse will try to infer numbers from string fields. It can be useful if JSON data contains quoted UInt64 numbers.
Disabled by default.
input_format_json_use_string_type_for_ambiguous_paths_in_named_tuples_inference_from_objects
Type: Bool
Default value: 0
Use String type instead of an exception in case of ambiguous paths in JSON objects during named tuples inference
input_format_json_validate_types_from_metadata
Type: Bool
Default value: 1
For JSON/JSONCompact/JSONColumnsWithMetadata input formats, if this setting is set to 1, the types from metadata in input data will be compared with the types of the corresponding columns from the table.
Enabled by default.
input_format_max_bytes_to_read_for_schema_inference
Type: UInt64
Default value: 33554432
The maximum amount of data in bytes to read for automatic schema inference.
input_format_max_rows_to_read_for_schema_inference
Type: UInt64
Default value: 25000
The maximum rows of data to read for automatic schema inference.
input_format_msgpack_number_of_columns
Type: UInt64
Default value: 0
The number of columns in inserted MsgPack data. Used for automatic schema inference from data.
input_format_mysql_dump_map_column_names
Type: Bool
Default value: 1
Match columns from table in MySQL dump and columns from ClickHouse table by names
input_format_mysql_dump_table_name
Type: String
Default value:
Name of the table in MySQL dump from which to read data