Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 636 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.1928. Min Range: -0.1928

Max Act: 0.1928. Min Act: -0.0000

Data Index: 1258174 (C4 (Web Text))

Max Activating Token Index: 597

Text #1

Max Range: 0.1928. Min Range: -0.1928

Max Act: 0.1528. Min Act: -0.0000

Data Index: 588481 (C4 (Web Text))

Max Activating Token Index: 1009

Text #2

Max Range: 0.1928. Min Range: -0.1928

Max Act: 0.1533. Min Act: -0.0000

Data Index: 202613 (C4 (Web Text))

Max Activating Token Index: 425

Text #3

Max Range: 0.1928. Min Range: -0.1928

Max Act: 0.1530. Min Act: -0.0000

Data Index: 739943 (C4 (Web Text))

Max Activating Token Index: 291

Text #4

Max Range: 0.1928. Min Range: -0.1928

Max Act: 0.1505. Min Act: -0.0000

Data Index: 144259 (C4 (Web Text))

Max Activating Token Index: 1002

Text #5

Max Range: 0.1928. Min Range: -0.1928

Max Act: 0.1586. Min Act: -0.0000

Data Index: 151675 (C4 (Web Text))

Max Activating Token Index: 119

Text #6

Max Range: 0.1928. Min Range: -0.1928

Max Act: 0.1518. Min Act: -0.0000

Data Index: 342665 (C4 (Web Text))

Max Activating Token Index: 854

Text #7

Max Range: 0.1928. Min Range: -0.1928

Max Act: 0.1319. Min Act: -0.0000

Data Index: 934180 (C4 (Web Text))

Max Activating Token Index: 389

Text #8

Max Range: 0.1928. Min Range: -0.1928

Max Act: 0.1453. Min Act: -0.0000

Data Index: 1158427 (C4 (Web Text))

Max Activating Token Index: 917

Text #9

Max Range: 0.1928. Min Range: -0.1928

Max Act: 0.1411. Min Act: -0.0000

Data Index: 1175405 (C4 (Web Text))

Max Activating Token Index: 140

Text #10

Max Range: 0.1928. Min Range: -0.1928

Max Act: 0.1360. Min Act: -0.0000

Data Index: 672149 (C4 (Web Text))

Max Activating Token Index: 381

Text #11

Max Range: 0.1928. Min Range: -0.1928

Max Act: 0.1342. Min Act: -0.0000

Data Index: 914911 (C4 (Web Text))

Max Activating Token Index: 220

Text #12

Max Range: 0.1928. Min Range: -0.1928

Max Act: 0.1541. Min Act: -0.0000

Data Index: 525320 (C4 (Web Text))

Max Activating Token Index: 811

Text #13

Max Range: 0.1928. Min Range: -0.1928

Max Act: 0.1322. Min Act: -0.0000

Data Index: 1188200 (C4 (Web Text))

Max Activating Token Index: 480

Text #14

Max Range: 0.1928. Min Range: -0.1928

Max Act: 0.1368. Min Act: -0.0000

Data Index: 999095 (C4 (Web Text))

Max Activating Token Index: 467

Text #15

Max Range: 0.1928. Min Range: -0.1928

Max Act: 0.1301. Min Act: -0.0000

Data Index: 842345 (C4 (Web Text))

Max Activating Token Index: 1021

Text #16

Max Range: 0.1928. Min Range: -0.1928

Max Act: 0.1308. Min Act: -0.0000

Data Index: 477415 (C4 (Web Text))

Max Activating Token Index: 312

Text #17

Max Range: 0.1928. Min Range: -0.1928

Max Act: 0.1255. Min Act: -0.0000

Data Index: 1298374 (C4 (Web Text))

Max Activating Token Index: 491

Text #18

Max Range: 0.1928. Min Range: -0.1928

Max Act: 0.1231. Min Act: -0.0000

Data Index: 737111 (C4 (Web Text))

Max Activating Token Index: 538

Text #19

Max Range: 0.1928. Min Range: -0.1928

Max Act: 0.1226. Min Act: -0.0000

Data Index: 1279820 (C4 (Web Text))

Max Activating Token Index: 235