Skip to content

Commit 0a65f11

Browse files
dlevy-msft-sqldlevy-msft-sql
committed
feat: add -f codepage flag for input/output encoding
Implements the -f flag for specifying input/output file encoding: - Format: codepage | i:codepage[,o:codepage] | o:codepage[,i:codepage] - Use 65001 for UTF-8 - --list-codepages shows all supported encodings Windows uses native MultiByteToWideChar/WideCharToMultiByte APIs for codepages not in the golang.org/x/text registry.
1 parent ca107b8 commit 0a65f11

14 files changed

Lines changed: 1780 additions & 5 deletions

File tree

‎COPILOT_CONTEXT.md‎

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
# Copilot Session Context - February 5, 2026
2+
3+
## User Preferences (from ~/.copilot-style.md)
4+
- Short, direct responses - no AI slop
5+
- Simplicity first, minimal impact
6+
- Plan before implementing
7+
- Squash commits to 1 per PR
8+
9+
## Repository
10+
- **Repo**: microsoft/go-sqlcmd (Go CLI for SQL Server)
11+
- **Fork**: dlevy-msft-sql/go-sqlcmd
12+
- **Default branch**: main
13+
14+
## PR Status Summary
15+
16+
### Ready to Merge (CI passing)
17+
| PR | Branch | Title |
18+
|----|--------|-------|
19+
| #701 | fix-chroma-tests | deps: bump chroma v2.23.1 and Go 1.24.13 |
20+
| #692 | feat/devcontainer | feat: add devcontainer for VS Code and GitHub Codespaces |
21+
| #637 | no-bom-flag | Add --no-bom flag for ODBC sqlcmd compatibility |
22+
| #639 | docs/fix-readme-freshness | docs: update README for documentation freshness |
23+
| #636 | exit-multiline | Implement multi-line EXIT(query) support |
24+
| #635 | r-default-fix | Allow -r without argument, defaulting to 0 |
25+
| #633 | lint-cleanup | Fix linting warnings |
26+
| #624 | feature/raw-error-messages | Add -j (--raw-errors) flag |
27+
28+
### May Need Attention
29+
| PR | Branch | Status |
30+
|----|--------|--------|
31+
| #688 | pr-685 | open vscode/ssms commands - check CI |
32+
| #638 | round-out-part2 | -f codepage flag - check CI |
33+
| #632 | perftrace | :perftrace and :help commands - check CI |
34+
| #631 | print-statistics | -p flag for statistics - check CI |
35+
| #630 | serverlist-command | :serverlist command - check CI |
36+
| #628 | regional-settings | -R and -f flags - check CI |
37+
| #626 | feature/allow-q-with-i | Allow -q with -i - check CI |
38+
39+
### External PRs
40+
| PR | Owner | Status |
41+
|----|-------|--------|
42+
| #693 | dependabot | CLOSE - superseded by #701 |
43+
| #682 | copilot-swe-agent | Waiting for Copilot to fix lint error |
44+
45+
## Common Lint Fixes Applied
46+
All PRs needed these patterns fixed:
47+
```go
48+
// Before (fails errcheck)
49+
defer os.Remove(file.Name())
50+
defer file.Close()
51+
52+
// After
53+
defer func() { _ = os.Remove(file.Name()) }()
54+
defer func() { _ = file.Close() }()
55+
```
56+
57+
## CI Notes
58+
- Go Vulnerability Check requires go1.24.13 (fixed in #701)
59+
- golangci-lint runs on PR changes only
60+
- Build runs tests against Linux SQL Server container
61+
62+
## Quick Commands
63+
```powershell
64+
# Check all your open PRs
65+
gh pr list --author dlevy-msft-sql --repo microsoft/go-sqlcmd
66+
67+
# Check CI status for a PR
68+
gh pr checks <PR#> --repo microsoft/go-sqlcmd
69+
70+
# Get lint errors from failed run
71+
gh run view <RUN_ID> --log-failed --repo microsoft/go-sqlcmd 2>&1 | Select-String "##\[error\]"
72+
73+
# Squash commits on a branch
74+
git fetch origin <branch>
75+
git checkout <branch>
76+
git reset --hard origin/<branch>
77+
git reset --soft $(git merge-base HEAD upstream/main)
78+
git commit -m "feat: description"
79+
git push origin <branch> --force-with-lease
80+
```
81+
82+
## Files Modified This Session
83+
- `.devcontainer/README.md` - Updated ARM64 guidance to recommend Codespaces
84+
- `.github/workflows/security.yml` - Updated go-version to 1.24.13
85+
- `pkg/sqlcmd/commands_test.go` - Various errcheck fixes
86+
- `pkg/sqlcmd/format_test.go` - Made colorizer test more flexible
87+
- `cmd/sqlcmd/sqlcmd_test.go` - Various errcheck fixes
88+
- `internal/tools/tool/tool_test.go` - Changed panic test to error test
89+
90+
## Personal Instructions Location
91+
Read at start of each session:
92+
```powershell
93+
Get-Content "C:/Users/DLevy/.copilot-style.md"
94+
```

‎README.md‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -151,6 +151,7 @@ The following switches have different behavior in this version of `sqlcmd` compa
151151
- To provide the value of the host name in the server certificate when using strict encryption, pass the host name with `-F`. Example: `-Ns -F myhost.domain.com`
152152
- More information about client/server encryption negotiation can be found at <https://docs.microsoft.com/openspecs/windows_protocols/ms-tds/60f56408-0188-4cd5-8b90-25c6f2423868>
153153
- `-u` The generated Unicode output file will have the UTF16 Little-Endian Byte-order mark (BOM) written to it.
154+
- `-f` Specifies the code page for input and output files. See [Code Page Support](#code-page-support) below for details and examples.
154155
- Some behaviors that were kept to maintain compatibility with `OSQL` may be changed, such as alignment of column headers for some data types.
155156
- All commands must fit on one line, even `EXIT`. Interactive mode will not check for open parentheses or quotes for commands and prompt for successive lines. The ODBC sqlcmd allows the query run by `EXIT(query)` to span multiple lines.
156157
- `-i` doesn't handle a comma `,` in a file name correctly unless the file name argument is triple quoted. For example:
@@ -255,6 +256,79 @@ To see a list of available styles along with colored syntax samples, use this co
255256
:list color
256257
```
257258

259+
### Code Page Support
260+
261+
The `-f` flag specifies the code page for reading input files and writing output. This is useful when working with SQL scripts saved in legacy encodings or when output needs to be in a specific encoding.
262+
263+
#### Format
264+
265+
```
266+
-f codepage # Set both input and output to the same codepage
267+
-f i:codepage # Set input codepage only
268+
-f o:codepage # Set output codepage only
269+
-f i:codepage,o:codepage # Set input and output to different codepages
270+
-f o:codepage,i:codepage # Same as above (order doesn't matter)
271+
```
272+
273+
#### Common Code Pages
274+
275+
| Code Page | Name | Description |
276+
|-----------|------|-------------|
277+
| 65001 | UTF-8 | Unicode (UTF-8) - default for most modern systems |
278+
| 1200 | UTF-16LE | Unicode (UTF-16 Little-Endian) |
279+
| 1201 | UTF-16BE | Unicode (UTF-16 Big-Endian) |
280+
| 1252 | Windows-1252 | Western European (Windows) |
281+
| 932 | Shift_JIS | Japanese |
282+
| 936 | GBK | Chinese Simplified |
283+
| 949 | EUC-KR | Korean |
284+
| 950 | Big5 | Chinese Traditional |
285+
| 437 | CP437 | OEM United States (DOS) |
286+
287+
#### Examples
288+
289+
**Run a script saved in Windows-1252 encoding:**
290+
```bash
291+
sqlcmd -S myserver -i legacy_script.sql -f 1252
292+
```
293+
294+
**Read UTF-16 input file and write UTF-8 output:**
295+
```bash
296+
sqlcmd -S myserver -i unicode_script.sql -o results.txt -f i:1200,o:65001
297+
```
298+
299+
**Process a Japanese Shift-JIS encoded script:**
300+
```bash
301+
sqlcmd -S myserver -i japanese_data.sql -f 932
302+
```
303+
304+
**Write output in Windows-1252 for legacy applications:**
305+
```bash
306+
sqlcmd -S myserver -Q "SELECT * FROM Products" -o report.txt -f o:1252
307+
```
308+
309+
**List all supported code pages:**
310+
```bash
311+
sqlcmd --list-codepages
312+
```
313+
314+
#### Notes
315+
316+
- When no `-f` flag is specified, sqlcmd auto-detects UTF-8/UTF-16LE/UTF-16BE BOM (Byte Order Mark) in input files and switches to the appropriate decoder. If no BOM is present, UTF-8 is assumed.
317+
- UTF-8 input files with BOM are handled automatically.
318+
- On Windows, additional codepages installed on the system are available via the Windows API, even if not shown by `--list-codepages`.
319+
- Use `--list-codepages` to see the built-in code pages with their names and descriptions.
320+
321+
#### Differences from ODBC sqlcmd
322+
323+
| Aspect | ODBC sqlcmd | go-sqlcmd |
324+
|--------|-------------|-----------|
325+
| **Default encoding (no BOM, no `-f`)** | Windows ANSI code page (locale-dependent, e.g., 1252) | UTF-8 |
326+
| **UTF-16 codepages (1200, 1201)** | Rejected by `IsValidCodePage()` API | Accepted |
327+
| **BOM detection** | Yes (UTF-8, UTF-16 LE/BE) | Yes (identical behavior) |
328+
| **`--list-codepages`** | Not available | Available |
329+
330+
**Migration note**: If you have UTF-8 encoded SQL scripts without a BOM that worked with ODBC sqlcmd on Windows, they should work identically or better with go-sqlcmd since go-sqlcmd defaults to UTF-8. However, if you have scripts in Windows ANSI encoding (e.g., Windows-1252) without a BOM, you may need to explicitly specify `-f 1252` with go-sqlcmd.
331+
258332
### Packages
259333

260334
#### sqlcmd executable

0 commit comments

Comments
 (0)