Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3035 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.3728. Min Range: -0.3728

Max Act: 0.3728. Min Act: -0.0000

Data Index: 694249 (C4 (Web Text))

Max Activating Token Index: 572

Text #1

Max Range: 0.3728. Min Range: -0.3728

Max Act: 0.3688. Min Act: -0.0000

Data Index: 293163 (C4 (Web Text))

Max Activating Token Index: 623

Text #2

Max Range: 0.3728. Min Range: -0.3728

Max Act: 0.3377. Min Act: -0.0000

Data Index: 1480940 (Python Code)

Max Activating Token Index: 531

Text #3

Max Range: 0.3728. Min Range: -0.3728

Max Act: 0.3005. Min Act: -0.0000

Data Index: 1615172 (Python Code)

Max Activating Token Index: 394

Text #4

Max Range: 0.3728. Min Range: -0.3728

Max Act: 0.3012. Min Act: -0.0000

Data Index: 1443290 (Python Code)

Max Activating Token Index: 373

Text #5

Max Range: 0.3728. Min Range: -0.3728

Max Act: 0.2992. Min Act: 0.0000

Data Index: 1064839 (C4 (Web Text))

Max Activating Token Index: 746

Text #6

Max Range: 0.3728. Min Range: -0.3728

Max Act: 0.2956. Min Act: 0.0001

Data Index: 759900 (C4 (Web Text))

Max Activating Token Index: 351

Text #7

Max Range: 0.3728. Min Range: -0.3728

Max Act: 0.2718. Min Act: -0.0000

Data Index: 263968 (C4 (Web Text))

Max Activating Token Index: 73

Text #8

Max Range: 0.3728. Min Range: -0.3728

Max Act: 0.2587. Min Act: -0.0000

Data Index: 361099 (C4 (Web Text))

Max Activating Token Index: 153

Text #9

Max Range: 0.3728. Min Range: -0.3728

Max Act: 0.2581. Min Act: 0.0000

Data Index: 252606 (C4 (Web Text))

Max Activating Token Index: 335

Text #10

Max Range: 0.3728. Min Range: -0.3728

Max Act: 0.2558. Min Act: -0.0000

Data Index: 1586882 (Python Code)

Max Activating Token Index: 747

Text #11

Max Range: 0.3728. Min Range: -0.3728

Max Act: 0.2274. Min Act: -0.0000

Data Index: 932535 (C4 (Web Text))

Max Activating Token Index: 943

Text #12

Max Range: 0.3728. Min Range: -0.3728

Max Act: 0.2481. Min Act: -0.0000

Data Index: 1617327 (Python Code)

Max Activating Token Index: 910

Text #13

Max Range: 0.3728. Min Range: -0.3728

Max Act: 0.2336. Min Act: -0.0000

Data Index: 1085693 (C4 (Web Text))

Max Activating Token Index: 689

Text #14

Max Range: 0.3728. Min Range: -0.3728

Max Act: 0.2439. Min Act: -0.0000

Data Index: 1213981 (C4 (Web Text))

Max Activating Token Index: 410

Text #15

Max Range: 0.3728. Min Range: -0.3728

Max Act: 0.2487. Min Act: -0.0000

Data Index: 1375990 (Python Code)

Max Activating Token Index: 276

Text #16

Max Range: 0.3728. Min Range: -0.3728

Max Act: 0.2180. Min Act: -0.0000

Data Index: 151705 (C4 (Web Text))

Max Activating Token Index: 273

Text #17

Max Range: 0.3728. Min Range: -0.3728

Max Act: 0.2432. Min Act: -0.0000

Data Index: 404658 (C4 (Web Text))

Max Activating Token Index: 322

Text #18

Max Range: 0.3728. Min Range: -0.3728

Max Act: 0.2399. Min Act: -0.0000

Data Index: 728129 (C4 (Web Text))

Max Activating Token Index: 155

Text #19

Max Range: 0.3728. Min Range: -0.3728

Max Act: 0.2546. Min Act: -0.0000

Data Index: 1276170 (C4 (Web Text))

Max Activating Token Index: 438