Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2543 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 0.4280. Min Range: -0.4280

Max Act: 0.3653. Min Act: 0.0000

Data Index: 1365626 (Python Code)

Max Activating Token Index: 336

Text #1

Max Range: 0.4280. Min Range: -0.4280

Max Act: 0.4096. Min Act: -0.0001

Data Index: 1631818 (Python Code)

Max Activating Token Index: 184

Text #2

Max Range: 0.4280. Min Range: -0.4280

Max Act: 0.4280. Min Act: -0.0001

Data Index: 1656508 (Python Code)

Max Activating Token Index: 926

Text #3

Max Range: 0.4280. Min Range: -0.4280

Max Act: 0.3918. Min Act: 0.0000

Data Index: 1566517 (Python Code)

Max Activating Token Index: 307

Text #4

Max Range: 0.4280. Min Range: -0.4280

Max Act: 0.3788. Min Act: 0.0000

Data Index: 1481423 (Python Code)

Max Activating Token Index: 510

Text #5

Max Range: 0.4280. Min Range: -0.4280

Max Act: 0.3867. Min Act: -0.0001

Data Index: 1630557 (Python Code)

Max Activating Token Index: 143

Text #6

Max Range: 0.4280. Min Range: -0.4280

Max Act: 0.3890. Min Act: 0.0000

Data Index: 1655523 (Python Code)

Max Activating Token Index: 583

Text #7

Max Range: 0.4280. Min Range: -0.4280

Max Act: 0.3998. Min Act: 0.0000

Data Index: 1510008 (Python Code)

Max Activating Token Index: 601

Text #8

Max Range: 0.4280. Min Range: -0.4280

Max Act: 0.3587. Min Act: 0.0000

Data Index: 1510198 (Python Code)

Max Activating Token Index: 333

Text #9

Max Range: 0.4280. Min Range: -0.4280

Max Act: 0.4011. Min Act: 0.0000

Data Index: 1575571 (Python Code)

Max Activating Token Index: 275

Text #10

Max Range: 0.4280. Min Range: -0.4280

Max Act: 0.3561. Min Act: 0.0000

Data Index: 1484631 (Python Code)

Max Activating Token Index: 788

Text #11

Max Range: 0.4280. Min Range: -0.4280

Max Act: 0.3692. Min Act: 0.0000

Data Index: 1491668 (Python Code)

Max Activating Token Index: 406

Text #12

Max Range: 0.4280. Min Range: -0.4280

Max Act: 0.3167. Min Act: -0.0000

Data Index: 1427073 (Python Code)

Max Activating Token Index: 167

Text #13

Max Range: 0.4280. Min Range: -0.4280

Max Act: 0.3389. Min Act: 0.0000

Data Index: 1545259 (Python Code)

Max Activating Token Index: 748

Text #14

Max Range: 0.4280. Min Range: -0.4280

Max Act: 0.3280. Min Act: -0.0001

Data Index: 1647434 (Python Code)

Max Activating Token Index: 43

Text #15

Max Range: 0.4280. Min Range: -0.4280

Max Act: 0.3269. Min Act: 0.0000

Data Index: 1512396 (Python Code)

Max Activating Token Index: 414

Text #16

Max Range: 0.4280. Min Range: -0.4280

Max Act: 0.2984. Min Act: 0.0000

Data Index: 1544684 (Python Code)

Max Activating Token Index: 436

Text #17

Max Range: 0.4280. Min Range: -0.4280

Max Act: 0.3276. Min Act: 0.0000

Data Index: 1520881 (Python Code)

Max Activating Token Index: 730

Text #18

Max Range: 0.4280. Min Range: -0.4280

Max Act: 0.3596. Min Act: 0.0000

Data Index: 1492560 (Python Code)

Max Activating Token Index: 722

Text #19

Max Range: 0.4280. Min Range: -0.4280

Max Act: 0.3240. Min Act: 0.0000

Data Index: 1516259 (Python Code)

Max Activating Token Index: 170