Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1908 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 2.6632. Min Range: -2.6632

Max Act: 2.5267. Min Act: -0.0000

Data Index: 1091208 (C4 (Web Text))

Max Activating Token Index: 831

Text #1

Max Range: 2.6632. Min Range: -2.6632

Max Act: 2.5484. Min Act: -0.0000

Data Index: 276433 (C4 (Web Text))

Max Activating Token Index: 450

Text #2

Max Range: 2.6632. Min Range: -2.6632

Max Act: 2.6632. Min Act: -0.0000

Data Index: 1124545 (C4 (Web Text))

Max Activating Token Index: 205

Text #3

Max Range: 2.6632. Min Range: -2.6632

Max Act: 2.3976. Min Act: -0.0000

Data Index: 979804 (C4 (Web Text))

Max Activating Token Index: 259

Text #4

Max Range: 2.6632. Min Range: -2.6632

Max Act: 2.1515. Min Act: -0.0000

Data Index: 128276 (C4 (Web Text))

Max Activating Token Index: 187

Text #5

Max Range: 2.6632. Min Range: -2.6632

Max Act: 1.5514. Min Act: -0.0000

Data Index: 620726 (C4 (Web Text))

Max Activating Token Index: 287

Text #6

Max Range: 2.6632. Min Range: -2.6632

Max Act: 1.4959. Min Act: -0.0000

Data Index: 1195893 (C4 (Web Text))

Max Activating Token Index: 289

Text #7

Max Range: 2.6632. Min Range: -2.6632

Max Act: 1.2710. Min Act: -0.0000

Data Index: 377175 (C4 (Web Text))

Max Activating Token Index: 284

Text #8

Max Range: 2.6632. Min Range: -2.6632

Max Act: 1.0768. Min Act: -0.0000

Data Index: 1605669 (Python Code)

Max Activating Token Index: 445

Text #9

Max Range: 2.6632. Min Range: -2.6632

Max Act: 0.9442. Min Act: -0.0001

Data Index: 1591513 (Python Code)

Max Activating Token Index: 688

Text #10

Max Range: 2.6632. Min Range: -2.6632

Max Act: 1.0283. Min Act: 0.0000

Data Index: 202885 (C4 (Web Text))

Max Activating Token Index: 169

Text #11

Max Range: 2.6632. Min Range: -2.6632

Max Act: 0.9704. Min Act: -0.0000

Data Index: 853319 (C4 (Web Text))

Max Activating Token Index: 719

Text #12

Max Range: 2.6632. Min Range: -2.6632

Max Act: 0.9188. Min Act: -0.0001

Data Index: 1614149 (Python Code)

Max Activating Token Index: 422

Text #13

Max Range: 2.6632. Min Range: -2.6632

Max Act: 0.8839. Min Act: -0.0000

Data Index: 1426828 (Python Code)

Max Activating Token Index: 737

Text #14

Max Range: 2.6632. Min Range: -2.6632

Max Act: 0.8853. Min Act: -0.0000

Data Index: 1630866 (Python Code)

Max Activating Token Index: 648

Text #15

Max Range: 2.6632. Min Range: -2.6632

Max Act: 0.8762. Min Act: -0.0001

Data Index: 1143212 (C4 (Web Text))

Max Activating Token Index: 513

Text #16

Max Range: 2.6632. Min Range: -2.6632

Max Act: 0.9002. Min Act: 0.0000

Data Index: 278268 (C4 (Web Text))

Max Activating Token Index: 398

Text #17

Max Range: 2.6632. Min Range: -2.6632

Max Act: 0.8348. Min Act: -0.0000

Data Index: 1606962 (Python Code)

Max Activating Token Index: 849

Text #18

Max Range: 2.6632. Min Range: -2.6632

Max Act: 0.8568. Min Act: 0.0001

Data Index: 248139 (C4 (Web Text))

Max Activating Token Index: 106

Text #19

Max Range: 2.6632. Min Range: -2.6632

Max Act: 0.9001. Min Act: -0.0000

Data Index: 217899 (C4 (Web Text))

Max Activating Token Index: 490