Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5481 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.3804. Min Range: -2.3804

Max Act: 2.3804. Min Act: -0.0000

Data Index: 46618 (C4 (Web Text))

Max Activating Token Index: 793

Text #1

Max Range: 2.3804. Min Range: -2.3804

Max Act: 2.1037. Min Act: -0.0000

Data Index: 463694 (C4 (Web Text))

Max Activating Token Index: 177

Text #2

Max Range: 2.3804. Min Range: -2.3804

Max Act: 2.1847. Min Act: -0.0000

Data Index: 662672 (C4 (Web Text))

Max Activating Token Index: 927

Text #3

Max Range: 2.3804. Min Range: -2.3804

Max Act: 2.0151. Min Act: -0.0000

Data Index: 1228040 (C4 (Web Text))

Max Activating Token Index: 778

Text #4

Max Range: 2.3804. Min Range: -2.3804

Max Act: 2.1656. Min Act: -0.0000

Data Index: 81029 (C4 (Web Text))

Max Activating Token Index: 145

Text #5

Max Range: 2.3804. Min Range: -2.3804

Max Act: 1.8529. Min Act: -0.0000

Data Index: 861307 (C4 (Web Text))

Max Activating Token Index: 643

Text #6

Max Range: 2.3804. Min Range: -2.3804

Max Act: 1.7559. Min Act: -0.0000

Data Index: 351150 (C4 (Web Text))

Max Activating Token Index: 617

Text #7

Max Range: 2.3804. Min Range: -2.3804

Max Act: 1.8240. Min Act: -0.0000

Data Index: 384270 (C4 (Web Text))

Max Activating Token Index: 198

Text #8

Max Range: 2.3804. Min Range: -2.3804

Max Act: 1.7624. Min Act: -0.0000

Data Index: 1166246 (C4 (Web Text))

Max Activating Token Index: 613

Text #9

Max Range: 2.3804. Min Range: -2.3804

Max Act: 1.6782. Min Act: -0.0000

Data Index: 37198 (C4 (Web Text))

Max Activating Token Index: 540

Text #10

Max Range: 2.3804. Min Range: -2.3804

Max Act: 1.7680. Min Act: -0.0000

Data Index: 1330985 (C4 (Web Text))

Max Activating Token Index: 555

Text #11

Max Range: 2.3804. Min Range: -2.3804

Max Act: 1.7432. Min Act: -0.0000

Data Index: 864163 (C4 (Web Text))

Max Activating Token Index: 472

Text #12

Max Range: 2.3804. Min Range: -2.3804

Max Act: 1.7563. Min Act: -0.0000

Data Index: 728919 (C4 (Web Text))

Max Activating Token Index: 430

Text #13

Max Range: 2.3804. Min Range: -2.3804

Max Act: 1.7406. Min Act: -0.0000

Data Index: 575957 (C4 (Web Text))

Max Activating Token Index: 514

Text #14

Max Range: 2.3804. Min Range: -2.3804

Max Act: 1.6326. Min Act: -0.0000

Data Index: 1236817 (C4 (Web Text))

Max Activating Token Index: 285

Text #15

Max Range: 2.3804. Min Range: -2.3804

Max Act: 1.6543. Min Act: -0.0000

Data Index: 132535 (C4 (Web Text))

Max Activating Token Index: 220

Text #16

Max Range: 2.3804. Min Range: -2.3804

Max Act: 1.8802. Min Act: -0.0000

Data Index: 242281 (C4 (Web Text))

Max Activating Token Index: 248

Text #17

Max Range: 2.3804. Min Range: -2.3804

Max Act: 1.6738. Min Act: -0.0000

Data Index: 759305 (C4 (Web Text))

Max Activating Token Index: 726

Text #18

Max Range: 2.3804. Min Range: -2.3804

Max Act: 1.6001. Min Act: -0.0000

Data Index: 1176979 (C4 (Web Text))

Max Activating Token Index: 1007

Text #19

Max Range: 2.3804. Min Range: -2.3804

Max Act: 1.6433. Min Act: -0.0000

Data Index: 133472 (C4 (Web Text))

Max Activating Token Index: 495