Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1408 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.1948. Min Range: -1.1948

Max Act: 1.1948. Min Act: -0.0000

Data Index: 119408 (C4 (Web Text))

Max Activating Token Index: 920

Text #1

Max Range: 1.1948. Min Range: -1.1948

Max Act: 0.9525. Min Act: -0.0000

Data Index: 1231767 (C4 (Web Text))

Max Activating Token Index: 812

Text #2

Max Range: 1.1948. Min Range: -1.1948

Max Act: 0.9884. Min Act: -0.0000

Data Index: 1231707 (C4 (Web Text))

Max Activating Token Index: 461

Text #3

Max Range: 1.1948. Min Range: -1.1948

Max Act: 0.8646. Min Act: -0.0000

Data Index: 399862 (C4 (Web Text))

Max Activating Token Index: 123

Text #4

Max Range: 1.1948. Min Range: -1.1948

Max Act: 1.0102. Min Act: -0.0000

Data Index: 385315 (C4 (Web Text))

Max Activating Token Index: 341

Text #5

Max Range: 1.1948. Min Range: -1.1948

Max Act: 0.9124. Min Act: -0.0000

Data Index: 185464 (C4 (Web Text))

Max Activating Token Index: 932

Text #6

Max Range: 1.1948. Min Range: -1.1948

Max Act: 0.9502. Min Act: -0.0000

Data Index: 253020 (C4 (Web Text))

Max Activating Token Index: 222

Text #7

Max Range: 1.1948. Min Range: -1.1948

Max Act: 0.8062. Min Act: -0.0000

Data Index: 714816 (C4 (Web Text))

Max Activating Token Index: 92

Text #8

Max Range: 1.1948. Min Range: -1.1948

Max Act: 0.8131. Min Act: -0.0000

Data Index: 532417 (C4 (Web Text))

Max Activating Token Index: 769

Text #9

Max Range: 1.1948. Min Range: -1.1948

Max Act: 0.8367. Min Act: -0.0000

Data Index: 232776 (C4 (Web Text))

Max Activating Token Index: 1021

Text #10

Max Range: 1.1948. Min Range: -1.1948

Max Act: 0.8667. Min Act: -0.0000

Data Index: 533844 (C4 (Web Text))

Max Activating Token Index: 586

Text #11

Max Range: 1.1948. Min Range: -1.1948

Max Act: 0.8159. Min Act: -0.0000

Data Index: 1644376 (Python Code)

Max Activating Token Index: 956

Text #12

Max Range: 1.1948. Min Range: -1.1948

Max Act: 0.8449. Min Act: -0.0000

Data Index: 1523244 (Python Code)

Max Activating Token Index: 935

Text #13

Max Range: 1.1948. Min Range: -1.1948

Max Act: 0.8519. Min Act: -0.0000

Data Index: 1236314 (C4 (Web Text))

Max Activating Token Index: 392

Text #14

Max Range: 1.1948. Min Range: -1.1948

Max Act: 0.7911. Min Act: -0.0000

Data Index: 1403668 (Python Code)

Max Activating Token Index: 507

Text #15

Max Range: 1.1948. Min Range: -1.1948

Max Act: 0.7465. Min Act: -0.0000

Data Index: 1591518 (Python Code)

Max Activating Token Index: 77

Text #16

Max Range: 1.1948. Min Range: -1.1948

Max Act: 0.7749. Min Act: -0.0000

Data Index: 1403497 (Python Code)

Max Activating Token Index: 78

Text #17

Max Range: 1.1948. Min Range: -1.1948

Max Act: 0.7377. Min Act: -0.0000

Data Index: 974080 (C4 (Web Text))

Max Activating Token Index: 327

Text #18

Max Range: 1.1948. Min Range: -1.1948

Max Act: 0.7785. Min Act: -0.0000

Data Index: 1581520 (Python Code)

Max Activating Token Index: 601

Text #19

Max Range: 1.1948. Min Range: -1.1948

Max Act: 0.7665. Min Act: -0.0000

Data Index: 1332799 (C4 (Web Text))

Max Activating Token Index: 225