Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1475 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 1.0981. Min Range: -1.0981

Max Act: 0.9675. Min Act: -0.0001

Data Index: 293852 (C4 (Web Text))

Max Activating Token Index: 370

Text #1

Max Range: 1.0981. Min Range: -1.0981

Max Act: 1.0981. Min Act: -0.0001

Data Index: 330766 (C4 (Web Text))

Max Activating Token Index: 498

Text #2

Max Range: 1.0981. Min Range: -1.0981

Max Act: 0.9323. Min Act: -0.0001

Data Index: 1212387 (C4 (Web Text))

Max Activating Token Index: 799

Text #3

Max Range: 1.0981. Min Range: -1.0981

Max Act: 1.0080. Min Act: -0.0001

Data Index: 1600651 (Python Code)

Max Activating Token Index: 575

Text #4

Max Range: 1.0981. Min Range: -1.0981

Max Act: 1.0247. Min Act: -0.0001

Data Index: 112750 (C4 (Web Text))

Max Activating Token Index: 390

Text #5

Max Range: 1.0981. Min Range: -1.0981

Max Act: 0.9445. Min Act: -0.0001

Data Index: 388735 (C4 (Web Text))

Max Activating Token Index: 578

Text #6

Max Range: 1.0981. Min Range: -1.0981

Max Act: 0.9375. Min Act: -0.0001

Data Index: 13555 (C4 (Web Text))

Max Activating Token Index: 226

Text #7

Max Range: 1.0981. Min Range: -1.0981

Max Act: 0.9177. Min Act: -0.0001

Data Index: 113838 (C4 (Web Text))

Max Activating Token Index: 542

Text #8

Max Range: 1.0981. Min Range: -1.0981

Max Act: 0.9075. Min Act: -0.0001

Data Index: 19003 (C4 (Web Text))

Max Activating Token Index: 326

Text #9

Max Range: 1.0981. Min Range: -1.0981

Max Act: 0.8977. Min Act: -0.0001

Data Index: 479563 (C4 (Web Text))

Max Activating Token Index: 756

Text #10

Max Range: 1.0981. Min Range: -1.0981

Max Act: 0.8933. Min Act: -0.0001

Data Index: 1158290 (C4 (Web Text))

Max Activating Token Index: 733

Text #11

Max Range: 1.0981. Min Range: -1.0981

Max Act: 0.8608. Min Act: -0.0001

Data Index: 1506564 (Python Code)

Max Activating Token Index: 568

Text #12

Max Range: 1.0981. Min Range: -1.0981

Max Act: 0.9183. Min Act: -0.0001

Data Index: 1001912 (C4 (Web Text))

Max Activating Token Index: 864

Text #13

Max Range: 1.0981. Min Range: -1.0981

Max Act: 0.8713. Min Act: -0.0001

Data Index: 1105782 (C4 (Web Text))

Max Activating Token Index: 358

Text #14

Max Range: 1.0981. Min Range: -1.0981

Max Act: 0.8118. Min Act: -0.0001

Data Index: 1508858 (Python Code)

Max Activating Token Index: 691

Text #15

Max Range: 1.0981. Min Range: -1.0981

Max Act: 0.8883. Min Act: -0.0001

Data Index: 70062 (C4 (Web Text))

Max Activating Token Index: 87

Text #16

Max Range: 1.0981. Min Range: -1.0981

Max Act: 0.8539. Min Act: -0.0001

Data Index: 430130 (C4 (Web Text))

Max Activating Token Index: 167

Text #17

Max Range: 1.0981. Min Range: -1.0981

Max Act: 0.8557. Min Act: -0.0001

Data Index: 1618146 (Python Code)

Max Activating Token Index: 1004

Text #18

Max Range: 1.0981. Min Range: -1.0981

Max Act: 0.8211. Min Act: -0.0001

Data Index: 1111099 (C4 (Web Text))

Max Activating Token Index: 37

Text #19

Max Range: 1.0981. Min Range: -1.0981

Max Act: 0.7915. Min Act: -0.0001

Data Index: 146713 (C4 (Web Text))

Max Activating Token Index: 34