Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2703 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.5576. Min Range: -2.5576

Max Act: 2.5576. Min Act: -0.0000

Data Index: 1655782 (Python Code)

Max Activating Token Index: 423

Text #1

Max Range: 2.5576. Min Range: -2.5576

Max Act: 2.2325. Min Act: -0.0000

Data Index: 1384674 (Python Code)

Max Activating Token Index: 480

Text #2

Max Range: 2.5576. Min Range: -2.5576

Max Act: 2.0876. Min Act: -0.0000

Data Index: 1361595 (Python Code)

Max Activating Token Index: 515

Text #3

Max Range: 2.5576. Min Range: -2.5576

Max Act: 2.0799. Min Act: -0.0000

Data Index: 1595485 (Python Code)

Max Activating Token Index: 855

Text #4

Max Range: 2.5576. Min Range: -2.5576

Max Act: 1.9296. Min Act: -0.0000

Data Index: 1411040 (Python Code)

Max Activating Token Index: 814

Text #5

Max Range: 2.5576. Min Range: -2.5576

Max Act: 2.1046. Min Act: -0.0000

Data Index: 1568951 (Python Code)

Max Activating Token Index: 808

Text #6

Max Range: 2.5576. Min Range: -2.5576

Max Act: 1.9767. Min Act: -0.0000

Data Index: 1542068 (Python Code)

Max Activating Token Index: 635

Text #7

Max Range: 2.5576. Min Range: -2.5576

Max Act: 1.8785. Min Act: -0.0000

Data Index: 1558947 (Python Code)

Max Activating Token Index: 735

Text #8

Max Range: 2.5576. Min Range: -2.5576

Max Act: 1.9838. Min Act: -0.0000

Data Index: 1651147 (Python Code)

Max Activating Token Index: 24

Text #9

Max Range: 2.5576. Min Range: -2.5576

Max Act: 1.8732. Min Act: -0.0000

Data Index: 1478946 (Python Code)

Max Activating Token Index: 755

Text #10

Max Range: 2.5576. Min Range: -2.5576

Max Act: 1.9763. Min Act: -0.0000

Data Index: 1539394 (Python Code)

Max Activating Token Index: 757

Text #11

Max Range: 2.5576. Min Range: -2.5576

Max Act: 1.9708. Min Act: -0.0000

Data Index: 1611721 (Python Code)

Max Activating Token Index: 44

Text #12

Max Range: 2.5576. Min Range: -2.5576

Max Act: 1.8547. Min Act: -0.0000

Data Index: 1404342 (Python Code)

Max Activating Token Index: 124

Text #13

Max Range: 2.5576. Min Range: -2.5576

Max Act: 1.9451. Min Act: -0.0000

Data Index: 1441776 (Python Code)

Max Activating Token Index: 596

Text #14

Max Range: 2.5576. Min Range: -2.5576

Max Act: 1.9140. Min Act: -0.0000

Data Index: 1656252 (Python Code)

Max Activating Token Index: 607

Text #15

Max Range: 2.5576. Min Range: -2.5576

Max Act: 2.0195. Min Act: -0.0000

Data Index: 1414745 (Python Code)

Max Activating Token Index: 346

Text #16

Max Range: 2.5576. Min Range: -2.5576

Max Act: 1.9122. Min Act: -0.0000

Data Index: 1458424 (Python Code)

Max Activating Token Index: 606

Text #17

Max Range: 2.5576. Min Range: -2.5576

Max Act: 1.7534. Min Act: -0.0000

Data Index: 1541477 (Python Code)

Max Activating Token Index: 608

Text #18

Max Range: 2.5576. Min Range: -2.5576

Max Act: 1.8769. Min Act: -0.0000

Data Index: 1464877 (Python Code)

Max Activating Token Index: 527

Text #19

Max Range: 2.5576. Min Range: -2.5576

Max Act: 1.8757. Min Act: -0.0000

Data Index: 1534476 (Python Code)

Max Activating Token Index: 461