Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 576 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.2548. Min Range: -1.2548

Max Act: 1.2548. Min Act: -0.0001

Data Index: 1648774 (Python Code)

Max Activating Token Index: 462

Text #1

Max Range: 1.2548. Min Range: -1.2548

Max Act: 1.2057. Min Act: -0.0001

Data Index: 1427882 (Python Code)

Max Activating Token Index: 170

Text #2

Max Range: 1.2548. Min Range: -1.2548

Max Act: 1.1371. Min Act: -0.0000

Data Index: 1515728 (Python Code)

Max Activating Token Index: 79

Text #3

Max Range: 1.2548. Min Range: -1.2548

Max Act: 1.1152. Min Act: -0.0000

Data Index: 1474112 (Python Code)

Max Activating Token Index: 834

Text #4

Max Range: 1.2548. Min Range: -1.2548

Max Act: 0.9351. Min Act: -0.0000

Data Index: 1651646 (Python Code)

Max Activating Token Index: 935

Text #5

Max Range: 1.2548. Min Range: -1.2548

Max Act: 0.9243. Min Act: -0.0000

Data Index: 1473140 (Python Code)

Max Activating Token Index: 498

Text #6

Max Range: 1.2548. Min Range: -1.2548

Max Act: 1.0194. Min Act: -0.0000

Data Index: 1590994 (Python Code)

Max Activating Token Index: 995

Text #7

Max Range: 1.2548. Min Range: -1.2548

Max Act: 0.9365. Min Act: -0.0000

Data Index: 1437254 (Python Code)

Max Activating Token Index: 255

Text #8

Max Range: 1.2548. Min Range: -1.2548

Max Act: 0.8769. Min Act: -0.0001

Data Index: 1540396 (Python Code)

Max Activating Token Index: 871

Text #9

Max Range: 1.2548. Min Range: -1.2548

Max Act: 0.9111. Min Act: -0.0000

Data Index: 1379999 (Python Code)

Max Activating Token Index: 201

Text #10

Max Range: 1.2548. Min Range: -1.2548

Max Act: 0.9293. Min Act: -0.0000

Data Index: 1473712 (Python Code)

Max Activating Token Index: 822

Text #11

Max Range: 1.2548. Min Range: -1.2548

Max Act: 0.8776. Min Act: -0.0000

Data Index: 1397841 (Python Code)

Max Activating Token Index: 38

Text #12

Max Range: 1.2548. Min Range: -1.2548

Max Act: 0.9331. Min Act: -0.0000

Data Index: 1460319 (Python Code)

Max Activating Token Index: 809

Text #13

Max Range: 1.2548. Min Range: -1.2548

Max Act: 0.8927. Min Act: -0.0000

Data Index: 1467179 (Python Code)

Max Activating Token Index: 356

Text #14

Max Range: 1.2548. Min Range: -1.2548

Max Act: 0.8375. Min Act: -0.0001

Data Index: 1508552 (Python Code)

Max Activating Token Index: 24

Text #15

Max Range: 1.2548. Min Range: -1.2548

Max Act: 0.8715. Min Act: -0.0000

Data Index: 1393359 (Python Code)

Max Activating Token Index: 179

Text #16

Max Range: 1.2548. Min Range: -1.2548

Max Act: 0.8643. Min Act: -0.0000

Data Index: 1451118 (Python Code)

Max Activating Token Index: 24

Text #17

Max Range: 1.2548. Min Range: -1.2548

Max Act: 0.8590. Min Act: -0.0000

Data Index: 1523651 (Python Code)

Max Activating Token Index: 591

Text #18

Max Range: 1.2548. Min Range: -1.2548

Max Act: 0.8895. Min Act: -0.0000

Data Index: 1456558 (Python Code)

Max Activating Token Index: 106

Text #19

Max Range: 1.2548. Min Range: -1.2548

Max Act: 0.8046. Min Act: -0.0000

Data Index: 1480085 (Python Code)

Max Activating Token Index: 57